Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)
Upload

Reply to this post | See parent post | Go Back
View Post [edit]

Poster: nick.clark Date: Jul 19, 2013 10:09am
Forum: web Subject: Re: Why can't I access old crawled sites who have recent robot.txt exclusions?

I am wondering the same thing. Is it possible that it's just a waiting game to see a new robots.txt file? I was in the process of rebuilding a site and normally I turn off robots while I am building. How soon does the IA scan the domain to put the archive back online?